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(54) Video coding and multiplexing 



(57) In multiplexing a plurality of coded data, de- 
grees of difficulty of coding of the data are measured 
(500V.500A), the coding rate for each of the data is de- 
cided (506) by allotting a certain target bit rate with a 
ratio of the each of the measured degrees of difficulty of 
coding so that the bit rate of the multiplexed data be- 



comes equal to the target bit rate, the data is then coded 
(306V,306A) in response to the decided coding rates, 
and the coded data is multiplexed (312) before being 
outputted to a transmission line (21 OA) or recording me- 
dium (21 0B). Thus, the bit rate of the outputted data is 
always kept at the target value to improve the usage rate 
of the transmission line. 
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of information coded at the coder is small) with the bit stream being used more than is necessary as a result. 

As shown in FIG. 4, a method is provided where each of the elementary streams are transmitted (or recorded) at 
variable bit rates. In this example, the widths of the respective graphs (corresponding to the bit rate) change in response 
to changes in the degree of difficulty of coding the data because the bit rate is allotted to each of the bit streams in 
s response to the amount of information for this data (a high bit rate is allotted when there is a large amount of information). 

An example of a configuration for a coding device for realizing this kind of transmission method is shown in FIG. 5. 

In FIG. 5, portions corresponding to portions in FIG. 2 are given the same numerals and the description is omitted 
as appropriate. 

In this example, the video data 200 - 1a to 200 - na is inputted to video coders 306V - 1 to 306V - n via rate 
10 measurers necessary for video 300V - 1 to 300V - n and the audio data 200 - 1 b to 200 - nb is inputted to audio coders 
306A - 1 to 306A - n via rate measurers necessary for audio 300A - 1 to 300A - n. 

The rate measurers necessary for video 300V - 1 to 300V - n and the rate measurers necessary for audio 300A - 
1 to 300A - n measures the rate (necessary rate) necessary during coding of the video data 200 la to 200 - na and 
the audio data 200 - 1b to 200 - nb using the tonality etc. showing undulations in the spectral distribution of the input 
is signal. The coding rates of the video coders 306V - 1 to 306V - n and the audio coders 306A - 1 to 306A - n are then 
controlled independently based on the measured data. 

At this time, it is necessary to ensure that the total for the bit rates for all of the data does not exceed the maximum 
bit rate for the transfer medium 21 OA. The maximum bit rate is then allotted to each of the items of data and coding is 
then carried out so that this maximum bit rate is not exceeded. In this way the total for the maximum bit rate allotted 
20 to each item of data becomes larger than the maximum bit rate of the transmission path but practical problems are few 
because the probability of a large number of data items of a high degree of coding difficulty occurring at the same time 
is small. 

The video coders 306V - 1 to 306V - n and the audio coders 306A - 1 to 306A - n code the video data 200 - la to 
200 - na and the audio data 200 - 1 b to 200 - nb at a prescribed bit rate based on the control of the rate measurers 

25 necessary for video 300V - 1 to 300V - n and the rate measurers necessary for audio 300A - 1 to 300A - n. The coded 
items of data and the coded rate information are supplied to the multiplexer 312, time coded based on each item of 
coding rate information and outputted to the transfer medium 21 OA (or the recording medium 21 0B). 

In this example, there is little noticeable deterioration or redundant consumption of the stream because the bit 
rates are allotted in response to the amount of information for each elementary stream. 

30 However, in the aforementioned example, coding is carried out at each of the coders independently. It is therefore 

possible that the coded data will exceed the maximum bit rate for the transfer medium 21 OA when a number of items 
of data for which the degree of difficulty of coding is very high occur, even though the probability of this is low. 

In order to prevent this, making the aforementioned maximum, bit. rate value allotted to each of the items of data 
small has been considered. However, if the maximum bit rate is made small, the quality of data for which a bit rate 

35 which exceeds this maximum bit rate is necessary falls. 

On the other hand, the multiplexing rate for the multiplexed data becomes extremely small with respect to the 
maximum bit rate of the transfer medium 21 OA when a large amount of data for which the degree of difficulty of coding 
is extremely low appears so that coding can be carried out at each of the coders independently. In this case, the transfer 
capability of the transfer medium is not used to it's maximum limit and redundancy therefore occurs. 

40 in order to resolve the aforementioned situation, the present invention is such that the bit rate for each item of data 

does not exceed the maximum bit rate of the transfer medium 21 OA or the recording medium 21 0B when a plurality of 
data is transmitted or recorded and the allotting of the most appropriate bit rates to each item of data is possible. 

Therefore, according to the present invention, a coding device for coding a plurality of data using one or more 
prescribed coding systems, multiplexing, and then outputting a plurality of the coded data comprises a measurer, a 

45 decider, an encoder and a multiplexer The measurer is for measuring degrees of difficulty of coding a plurality of the 
data. The decider is for deciding coding rates for a plurality of the data in accordance with a target bit rate and the 
degrees of difficulty of coding for a plurality of the data in such a manner that the bit rate of the output data becomes 
equal to the target bit rate. The encoder is for coding a plurality of the data in response to each of the decided coding 
rates and the multiplexer is for multiplexing a plurality of the data coded by the encoder. 

so Further, according to the present invention, a coding method for coding a plurality of data, multiplexing, and then 

outputting a plurality of the coded data, comprises a measuring step, deciding step, coding step and multiplexing step. 
The measuring step is for measuring degrees of difficulty of coding a plurality of the data. The deciding step is for 
deciding coding rates for a plurality of the data in accordance with a target bit rate and the degrees of difficulty of coding 
for a plurality of the data in such a manner that the bit rate of the output data becomes equal to the target bit rate. The 

ss coding step is for coding a plurality of the data in response to each of the decided coding rates. The multiplexing step 
is for multiplexing a plurality of the data coded in the coding step. 

Moreover, according to the present invention, a transmission device for coding a plurality of data using one or 
more prescribed coding systems, multiplexing, and then transmitting a plurality of the coded data comprises a measurer, 




BNSDOCID:<EP 0784409A2> 



* 



EP 0 784 409 A2 



10 



15 



20 



25 



30 



35 



40 



45 



■MB! 

XpW*. data. ^ _ mnMr , devic . and emission m«hod of ,he pr eS en, uwan.on, 

l„ Ih. coding <»* 9 ' *X 0 W«*o *» ^ d can ,u,lher a 

mmmssm 
mmmmm 

occurring in the examp.e o« FIG ,2; . rates of each of the elementary streams when data 

FIG 4 is a view showing changes with respect io 

measurer of FIG. 6; and 



BNSDOClD*<EP 0784409A2> 



EP 0 784 409 A2 

FIG. 1 2 is a block diagram showing an example of a still further configuration of the video coding degree of difficulty 
measurer of FIG. 6. 

FIG. 6 is a block view of an example configuration of a coding device of the present invention. In FIG. 6, portions 
s corresponding to portions in FIG. 5 are given the same numerals and their description is omitted. In this embodiment, 
video coding difficulty measurers 500V - 1 to 500V - n (measuring means), audio coding difficulty measurers 500A - 1 
to 500A - n (measuring means) and a coding rate allocator 506 (control means) are added to the example in FIG. 5. 
Other aspects of the configuration are the same as the case in FIG. 5. 

The video coding difficulty measurers 500V - 1 to 500V - n measure the difficulty of coding the video data 200 - 
10 1a to 200 - na. The audio coding difficulty measurers 500A - 1 to 500A - n measure the difficulty of coding 200 - 1b to 
200 - nb. 

The coding rate allocator 506 decides the bit rates allotted to each of the elementary streams in response to the 
degrees of difficulty of coding provided from the video coding difficulty measurers 500V - 1 to 500V - n and the audio 
coding difficulty measurers 500A - 1 to 500A - n. The decided bit rates are then supplied to video coders 306V - 1 to 
is 306V - n (coding means) and audio coders 306A - 1 to 306A - n (coding means). 

FIG. 7 is a block diagram showing an example of the detailed configuration of the video coding difficulty measurer 
500V - 1 (the remaining video coding difficulty measurers 500V - 2 to 500V - n, and the audio coding difficulty measurers 
500 A - 1 to 500A - n also have the same configuration)., 

In FIG. 7, a delay part 600 delays (stores) the inputted video data 200 - la by just the GOP (Group of Pictures) 
20 portion and then outputs this data. A coder 601 quantizes the inputted video data 200 - 1a in prescribed quantization 
steps (for example, Stepsize = 2) and then outputs this data. An coded volume calculator 602 adds up the coded 
amounts of data (data volumes) outputted from the coder 601 and outputs this as the degree of difficulty of coding. 

Next, the operation of the video coding degree of difficulty measurer the embodiment will be described. 

Video data 200 - la to 200 - na and audio data 200 - 1b to 200 - nb comprising programs 200 - 1 to 200 - n is 
25 supplied to the video coding difficulty measurers 500V - 1 to 500V - n and the audio coding difficulty measurers 500A 
- 1 to 500A - n. 

The inputted video data 200 - 1a is delayed by one GOP period at the video coding difficulty measurer 500V - 1 
shown in FIG. 7 and outputted. These signals are then supplied to the video coder 306V - 1 . 

Data supplied to the coder 601 is coded in quantization steps of, for example, Stepsize = 2, with the coded data 
30 then being supplied to the coded volume calculator 602. The coded volume calculator 602 then adds up the coded 
volumes for the inputted data across the GOP and these results are outputted. 

When the first data for the GOP is inputted at the coded volume calculator 602 calculations commence and these 
calculations then finish when the data for the last GOP is inputted, with the calculation results then being outputted i. 
e. it takes the time for 1 GOP until calculation results are outputted. The delay part 600 then delays the video data 200 
35 - la by one GOP portion and the timing is adjusted. 

Further, an example of the detailed configuration of the audio coding difficulty measurer 500A - 1 (the configuration 
is the same for the other audio coding difficulty measurers 500A - 2 to 500A - n) is also described using FIG. 7. The 
audio coding difficulty measurer 500A - 1 also basically comprises a delay part 605, a coder 606 and a coding amount 
calculator 607 but as the internal construction of each of the parts differs to those for the video coding difficulty meas- 
40 urers, the reference numerals for each part are changed and are described in detail in the following. 

In FIG. 7 the delay part 605 delays (accumulates) the inputted audio data 200 - 1b by just an audio frame portion 
and then outputs this data. This audio frame comprises a set of samples of about, for example, 20 to 30 msecs. The 
coder 606 quantizes the inputted audio data 200 - 1b in prescribed quantization steps (for example, Stepsize = 2) in 
such a manner as to output the coded data. The coding amount calculator 607 adds up the coding amount (data 
45 amount) for the coded data outputted from the coder 606 in audio frame units and outputs this as the degree of difficulty 
of coding. 

Next, a description is given of the operation of the embodiment of this audio coding difficulty measurer. 

The audio data 200 - 1b to 200 - nb comprising part of the program 200 - 1 to 200 - n is supplied to the audio 
coding difficulty measurers 500A - 1 to 500A - n. 
50 The inputted audio data 200 - 1 b is then delayed by just one audio frame portion at the delay part 605 of the audio 

coding difficulty measurer 500A - 1 shown in FIG. 2 and outputted. These signals are then supplied to the audio coder 
306A-1. 

The audio data supplied to the coder 606 is coded using, for example, quantization steps of Stepsize = 6, with the 
coded data then being supplied to the coding amount calculator 607. The coding amount calculator 607 then totals the 
55 amount of code for the inputted data across the audio frame and outputs the results. 

The coding amount calculator 607 starts calculations when the first coded data for the audio frame is inputted with 
the calculations ending when the coded data for the final audio frame is inputted and the calculation results being 
outputted i.e. the time until the calculation results are outputted is the time for one au6\o frame. The delay part 605 
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reference value PCR (Program Clock Reference) etc. 

Program editing information (PSI; Program Specific Information) showing the combinations etc. of the elementary 
streams comprising each program is also inserted as transport packets to be transported together with the other data 
while the transport stream (TS) comprised of the 188 byte length transport packets are being generated. 
5 Next, a description will be given with respect to the receiving side (decoding device). Multiplexed streams trans- 

mitted using the above data configuration are separated into elementary streams using the separating device shown 
in FIG. 9. The separating device shown in FIG. 9 is a virtual ideal decoder model and is referred to as a Target System 
Decoder for Transport (T-TSD). 

In FIG. 9, a switch 800 is a schematic representation of the separating circuit, with each of the elementary streams 
10 being separated and extracted from the inputted time multiplexed transport streams. Transport buffers (TB^ , TB n (where 
n shows the elementary stream number)) 801 and 802 absorb packet jitter. Buffers (B v B n ) 803 and 804 for system 
target decoding use are main buffers for storing and outputting prescribed data. 

The buffer 803 for the system target decoder is a buffer for video elementary stream use and can absorb 4 ms of 
jitter. Similarly, a buffer (BVBV) 805 is a buffer for video use that stores and outputs a prescribed amount of video data. 
is in particular, this buffer (BVBV) for elementary stream use is defined within standards relating to video signals within 
the MPEG system. 

A decoder (Dn) 806 is for decoding the elementary stream back to the original data. However, the decoder (Dn) 
806 decodes the video elementary stream other than that coded on the basis of the MPEG method or the audio ele- 
mentary stream back to the original video data or audio data. A video decoder 807 also decodes the video elementary 
20 stream back to the form of the original data. The video decoder 807 comprises a decoder 807a, a recorder buffer 807b 
for obtaining synchronization with other data and a changeover switch 807c. 

The inputted transport stream (refer to FIG. 8) inputted for this embodiment separates the respective elementary 
streams using the switch 800. Namely, the transport stream is separated into the video and audio elementary streams 
comprising this transport stream. The respective elementary streams are then supplied to the 51 2 byte transport buffers 
25 801 and 802. 

At this time, the elementary streams provided from the switch 800 to each of the transport buffers 801 and 802 
are separated and extracted using multiplexed streams that are single string bit streams. Data is therefore supplied 
intermittently (in bursts) from the switch 800 to each of the transport buffers 801 and 802. 

Each of the element streams outputted from the transport buffers 801 and 802 are inputted to system target decoder 
30 buffers 803 and 804 and outputted after a prescribed amount of stream data has been accumulated. 

Elementary streams outputted from the system target decoder buffer 803 are video elementary streams and are 
inputted to the buffer 805 for video elementary stream use. On the other hand, the elementary stream outputted from 
the system target decoder buffer 804 are inputted to the decoder 806, decoded to the original data and outputted. 

Video elementary streams outputted from the buffer 805 for video elementary stream use are inputted to the video 
35 decoder 807. The video elementary stream inputted to the video decoder 807 is decoded into the original video data 
by the decoder 807a. The decoded data is then supplied to the recorder buffer 807b and the changeover switch 807c, 
respectively. 

When the inputted multiplexed stream only comprises a video elementary stream (not including the case of an 
audio elementary stream), data outputted from the decoder 807a is directly outputted via the changeover switch 807c. 

40 Further, when the multiplexed stream comprises a video elementary stream and an audio elementary stream, it is 
necessary to obtain synchronization between the video data and the audio data. The output of the decoder 807a is 
therefore accumulated at the recorded buffer 807b and outputted via the changeover switch 807c after synchronization 
has been obtained with the audio data. Information for obtaining this synchronization can be referred to as a Presen- 
. tation Time Stamp (PTS) and this can be carried out by side information added to the multiplexed stream. 

45 When data is sent and received between a coding device on the data transmitting side and a decoding device on 

the data receiving side, it is necessary to ensure that the amount of data sent from the coding side does not exceed 
the amount of data that can be processed by the decoding device. In this embodiment, the coding device transmits 
data in such a manner that the buffers (transport buffers 801 , 802, system target decoder buffers 803 and 804 and the 
buffer 805 for video elementary stream use etc.) existing at the decoding device do not overflow. 

so At this time, the data rate (speed of transferring data) for transferring multiplexed streams from the transmitting 

side to the receiving side is referred to as the MUX_rate as described above, and is a transmission rate that is higher 
than the average rates of the elementary streams comprising all of the programs added together. 

FIG. 10 shows the situation where the GOP unit bit rates for elementary streams transmitted from the transmission 
side (coding device) to the receiving side (decoding device) change with respect to time. The horizontal axis of FIG. 

55 10 expresses the time T and the vertical axis shows the bit rates allotted in GOP units to the elementary streams. 
Further, the multiplexed control information is information showing the GOP unit bit rates of each of the elementary 
streams. 

Transmission rates are allotted to the video data 200 - la to 200 - na and the audio data 200 - 1b to 200 - nb in 
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data becomes of a fixed bit rate, with the combined data then being outputted via a transmission path. It is therefore 
possible to allot bit rates in accordance with the amount of information for each data item without the bit rate of the 
combined data exceeding the maximum bit rate for the transmission path. 

With the coding difficulty measurer of the embodiment of the present invention, the degree of difficulty of coding 
video is obtained in GOP units and the degree of difficulty of coding audio is obtained in audio frame units. The present 
invention is, however, by no means limited in this respect and any units may be used for obtaining the degree of difficulty 
of coding that do not change the scope of the present invention are possible. 

Further, the embodiment of the present invention is realized by the hardware shown in the block diagrams but the 
present invention is by no means limited in this respect and may also be realized by software using a CPU and/or 
memory etc. 

Moreover, various modifications and applications can be considered that do not deviate from the main points of 
the present invention. As a result, the main points of the present invention are by no means limited to the embodiment. 

Claims 

1. A coding device for coding a plurality of data using one or more prescribed coding systems, multiplexing, and then 
outputting a plurality of said data as being coded, said coding device comprising: measuring means for measuring 
degrees of difficulty of coding a plurality of said data; deciding means for deciding coding rates for a plurality of 
said data in accordance with a target bit rate and said degrees of difficulty of coding for a plurality of said data in 
such a manner that the bit rate of said data being outputted becomes equal to said target bit rate; coding means 
for coding a plurality of said data in response to each of said coding rates being decided; and multiplexing means 
for multiplexing a plurality of said data coded by said coding means. 

2. A coding device according to claim 1 , wherein a plurality of said data includes at least a plurality of video data. 

3. A coding device according to claim 2, wherein a plurality of said data further includes audio data. 

4. A coding device according to claim 1 , wherein said prescribed coding system is a Moving Picture Experts Group 
system. 

5. A coding device according to claim t , wherein said coding rate for each of a plurality of said data is decided by 
allotting said target bit rate with a ratio of said degree of difficulty of coding for said each of a plurality of said data 
to a cumulative addition of said degrees of difficulty of coding. 

6. A coding device according to claim 2, wherein said video data is measured by said measuring means with respect 
to said degrees of difficulty of coding in Group Of Picture unit. 

7. A coding device according to claim 2, wherein said video data provides said degree of difficulty of coding as an 
amount of codes when said video data is coded at a prescribed quantization step size. 

8. A coding device according to claim 2, wherein said video data provides said degree of difficulty of coding as a 
cumulative addition of squares of one of coding target pixel values and difference values of said coding target pixel 
values. -v...---. 

9. A coding device according to claim 2, wherein said video data provides said degree of difficulty of coding as a 
cumulative addition of one of coding target pixel values and absolute values of difference values of said coding 
target pixel values. - 

10. A coding device according to claim 3, wherein said audio data is measured by said measuring means with respect 
to said degree of difficulty of coding in audio frame unit. 

11. A coding device according to claim 3, wherein said audio data provides said degree of difficulty of coding as an 
amount of codes when said audio data is coded at prescribed quantization step sizes. 

12. A coding method for coding a plurality of data multiplexing, and then outputting a plurality of said data as being 
coded, said coding method comprising the steps of: 
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by allotting said target bit rate with a ratio of said degree of difficulty of coding for said each of a plurality of said 
data to a cumulative addition of said degrees of difficulty of coding. 

28. A transmission device according to claim 24, wherein said video data is measured by said measuring means with 
s respect to said degrees of difficulty of coding in Group Of Picture unit. 

29. A transmission device according to claim 24, wherein said video data provides said degree of difficulty of coding 
as an amount of codes when said video data is coded at a prescribed quantization step size. 

10 30. A transmission device according to claim 24, wherein said video data provides said degree of difficulty of coding 
as a cumulative addition of squares of one of coding target pixel values and difference values of said coding target 
pixel values. 

31. A transmission device according to claim 24, wherein said video data provides said degree of difficulty of coding 
is as a cumulative addition of one of coding target pixel values and absolute values of difference values of said coding 

target pixel values. 

32. A transmission device according to claim 25, wherein said audio data is measured by said measuring means with 
respect to said degree of difficulty of coding in audio frame unit. 

20 

33. A transmission device according to claim 25, wherein said audio data provides said degree of difficulty of coding 
as an amount of codes when said audio data is coded at prescribed quantization step sizes. 

34. A transmission method for transmitting a plurality of said data which are coded and multiplexed before being 
25 transmitted, said transmission method comprising the steps of: 

measuring degrees of difficulty of coding a plurality of said data; 

deciding coding rates for a plurality of said data in accordance with a target bit rate and said degrees of difficulty 
of coding for a plurality of said data in such a manner that a bit rate of said data being outputted becomes 
30 equal to said target bit rate; 

coding a plurality of said data in response to each of said coding rates being decided; 
multiplexing a plurality of said data coded in said step of coding; and 
transmitting said data being multiplexed. 

35 35. A transmission method according to claim 34, wherein a plurality of said data includes at least a plurality of video 
data. 

36. A transmission method according to claim 35, wherein a plurality of said data further includes audio data. 

40 37. A transmission method according to claim 34, wherein said prescribed coding system is a Moving Picture Experts 
Group system. 

38. A transmission method according to claim 34, wherein said coding rate for each of a plurality of said data is decided 
by allotting said target bit rate with a ratio of said degree of difficulty of coding for said each of a plurality of said 

45 data to a cumulative addition of said degrees of difficulty of coding. 

39. A transmission method according to claim 34, wherein said video data is measured by said measuring means with 
respect to said degrees of difficulty of coding in Group Of Picture unit. 

so 40. A transmission method according to claim 34, wherein said video data provides said degree of difficulty of coding 
as an amount of codes when said video data is coded at a prescribed quantization step size. 

41. A transmission method according to claim 34, wherein said video data provides said degree of difficulty of coding 
as a cumulative addition of squares of one of coding target pixel values and difference values of said coding target 

55 pixel values. 

42. A transmission method according to claim 34, wherein said video data provides said degree of difficulty of coding 
as a cumulative addition of one of coding target pixel values and absolute values of difference values of said coding 
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target pixel values. 

43. A transmission method according to claim 35, wherein said audio data is measured by said measuring means with 
respect to said degree of difficulty of coding in audio frame unit. 



44. A transmission method according to claim 35, wherein said audio data provides said degree of difficulty of coding 
as an amount of codes when said audio data is coded at prescribed quantization step sizes. 

45. A method of providing a multiplexed data of a plurality of data for being recorded in a recording medium which 
stores coded data including said multiplexed data as being decodable 1 said method comprising the steps of: 

measuring degrees of difficulty of coding a plurality of said data; 

deciding coding rates for a plurality of said data in accordance with a target bit rate and said degrees of difficulty 
of coding for a plurality of said data in such a manner that a bit rate of said data to be recorded in said recording 
• medium becomes equal to said target bit rate; 
coding a plurality of said data in accordance with a prescribed coding system in response to each of said 
coding rate being decided; 

multiplexing a plurality of said data coded in said step of coding; and 
outputting said multiplexed data for being recorded in said recording medium. 
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